Packet capture platforms generally operate as follows: packets are ingested real-time through either a network tap port being fed by an attached data center switch, or a promiscuous mode-set network adapter listening on a backbone which carries traffic of interest. Packets are stored in a locally attached repository. The local repository includes a file system that stores raw packets, as well as a juxtaposed index that allows for retrieval of those packets. The packet repository may be indexed by one or more parameters or elements. When the packet repository is filled to its capacity with raw packets, it will roll-over on the storage and write over the least recently stored packets with newly captured content. The indexes of the associated packets that have been overwritten are removed. This is therefore an inherently circular packet repository, with “circular storage sizing” over “sustained rate of captured traffic” defining a window of time that packets can be retrieved for analytics.
Packets are retrieved by a search process, initiated by an external request that uses the packet index to select packets. The retrieved packets may be assembled locally to be copied to another platform for analytics or sent via a network connection to some other requesting application.
Some traditional packet capture platforms may attach the platform to networked attached storage to extend the window of packet availability for analytics. In this case, the indexes for the remote storage are stored on the capture platform. When network attached storage fills to capacity, it will roll over and write over the least recently stored packets starting with the locally attached repository. The indexes of the associated packets that have been overwritten are also removed.
Packet capture to disk in real-time is an expensive operation and requires that the capture process is a high priority process elevated above the priority for processing of incoming search requests. This assures that all packets are written to disk with zero packet loss. When there are available resources for search, resources will be allocated to the search process to fulfill requests. A delicate balance is required to assure zero packet loss and thus search requests can be delayed for a considerable amount of time.
Retrieval speeds for packets are typically slow because resources for search can be starved by the packet capture process prioritization. Extending the packet availability window through network attached storage linearly decreases the packet retrieval speeds. On the other hand, the search process is singularly located on the capture platform and does not scale with increased storage. A typical packet retrieval request may take anywhere from 30 seconds for a very small amount of time window to hours for larger time windows at maximal capture rate and volume. The retrieval speed increases proportionately with the amount of extended storage and places further burden on a search process that is already vulnerable to resource starvation and access latency.
Another problem associated with this architecture includes latency, specifically associated with a shared network writing to shared disk repositories. The search process now competes with unknown resources for network bandwidth and controller and disk write request servicing resources. Moreover, the network path to the disk repository becomes a potential single point of failure, and the system cannot be upgraded without taking the packet capture platform off-line. Additionally, in applications involving secure packet content attribute transaction or any application deployed in a secure packet capture environment, the packets must be encrypted as they travel over a shared network.
Thus, there is a need for an improved packet capture platform to address the noted deficiencies.